{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/pinecone-io/examples/blob/master/learn/experimental/semantic-search-intro/jaccard.ipynb) [![Open nbviewer](https://raw.githubusercontent.com/pinecone-io/examples/master/assets/nbviewer-shield.svg)](https://nbviewer.org/github/pinecone-io/examples/blob/master/learn/experimental/semantic-search-intro/jaccard.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Similarity Methods\n",
    "\n",
    "## Traditional\n",
    "\n",
    "### Jaccard\n",
    "\n",
    "The Jaccard metric can be calculated as the number of shared instances between two sets of data, over the total number of instances between both sets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "a = \"his thought process was on so many levels that he gave himself a phobia of heights\"\n",
    "b = \"there is an art to getting your way and throwing bananas on to the street is not it\"\n",
    "c = \"it is not often you find soggy bananas on the street\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will first need to split our sentences into lists."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['there', 'is', 'an', 'art', 'to', 'getting', 'your', 'way', 'and', 'throwing', 'bananas', 'on', 'to', 'the', 'street', 'is', 'not', 'it']\n"
     ]
    }
   ],
   "source": [
    "a = a.split()\n",
    "b = b.split()\n",
    "c = c.split()\n",
    "print(b)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And convert our lists into sets (removing any duplication of words)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'an', 'bananas', 'the', 'getting', 'it', 'on', 'street', 'way', 'your', 'there', 'throwing', 'not', 'and', 'is', 'art', 'to'}\n"
     ]
    }
   ],
   "source": [
    "a = set(a)\n",
    "b = set(b)\n",
    "c = set(c)\n",
    "print(b)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We then calculate the total number of shared words between sets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'on'}\n",
      "1\n"
     ]
    }
   ],
   "source": [
    "shared_a_b = a.intersection(b)\n",
    "print(shared_a_b)\n",
    "print(len(shared_a_b))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And then we calculate the total number of *unique* words between each set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "31\n"
     ]
    }
   ],
   "source": [
    "count_a_b = len(a.union(b))\n",
    "print(count_a_b)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To calculate the Jaccard similarity between both, we take the number of shared words / total words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.03225806451612903"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(shared_a_b) / count_a_b"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And there is our Jaccard similarity - we can rewrite this into a function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "def jac(x: set, y: set):\n",
    "    shared = x.intersection(y)  # selects shared tokens only\n",
    "    return len(shared) / len(x.union(y))  # union adds both sets together"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.03225806451612903"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "jac(a, b)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.35"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "jac(b, c)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And we find that sentences `b` and `c` score much better, as we would expect. Now, it isn't perfect, two sentences that share nothing but words like *'the'*, *'a'*, *'how'*, etc - could return high Jaccard scores despite being completely dissimilar. We can go part way to solving some of these shortcomings through the use of stopword removal, stemming or lemmatization (so words like *'travelling'* and *'travels'* can match), and other preprocessing techniques. However, there are other methods that avoid these problems altogether.\n",
    "\n",
    "---\n",
    "\n",
    "## w-Shingling\n",
    "\n",
    "w-shingling is a very similar method to Jaccard similarity. It uses the exact same `intersection / union` logic, but with *'shingles'*. Let's put together an example using bigrams - eg 2-shingling."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "a = \"his thought process was on so many levels that he gave himself a phobia of heights\".split()\n",
    "b = \"there is an art to getting your way and throwing bananas on to the street is not it\".split()\n",
    "c = \"it is not often you find soggy bananas on the street\".split()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'he gave', 'gave himself', 'phobia of', 'on so', 'many levels', 'thought process', 'himself a', 'process was', 'a phobia', 'was on', 'his thought', 'that he', 'of heights', 'levels that', 'so many'}\n"
     ]
    }
   ],
   "source": [
    "a_shingle = set([' '.join([a[i], a[i+1]]) for i in range(len(a)) if i != len(a)-1])\n",
    "b_shingle = set([' '.join([b[i], b[i+1]]) for i in range(len(b)) if i != len(b)-1])\n",
    "c_shingle = set([' '.join([c[i], c[i+1]]) for i in range(len(c)) if i != len(c)-1])\n",
    "print(a_shingle)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.0"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "jac(a_shingle, b_shingle)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.125"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "jac(b_shingle, c_shingle)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'bananas on', 'is not', 'the street'}"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b_shingle.intersection(c_shingle)  # these are the matching shingles"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "base",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.13 (default, Mar 28 2022, 06:59:08) [MSC v.1916 64 bit (AMD64)]"
  },
  "orig_nbformat": 2,
  "vscode": {
   "interpreter": {
    "hash": "5fe10bf018ef3e697f9035d60bf60847932a12bface18908407fd371fe880db9"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
